feat: [AWS] OTel to ECS converter pipelines #15570

Kavindu-Dodan · 2025-10-06T19:51:35Z

Proposed commit message

This PR introduces OTel ECS converter integration for AWS signals.

The following table explains the supported signal type, source data set data must be ingested and the target data set converted data ends up,

Data source	OTel data set (source)	ECS data set (destination)
AWS ELB logs	aws.elbaccess.otel	aws.elb_logs
VPC flow logs	aws.vpcflow.otel	aws.vpcflow
CloudTrail logs	aws.cloudtrail.otel	aws.cloudtrail

The destination data set matches AWS Integration ¹ and supports dashboard assets of the signal.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Build and install the OTel ECS converter integration
Install AWS integration assets only to obtain necessary assets like dashboards and pipelines
Ingest OTel native data from one of the supported sources (you can use EDOT AWS ² for this)
Validate ingested data in destination data set & check ECS schema based dashboards

elastic-vault-github-plugin-prod · 2025-10-09T21:56:20Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

vigneshshanmugam · 2025-10-17T21:52:31Z

packages/otel_ecs_converter/manifest.yml

+owner:
+  github: elastic/obs-ds-hosted-services
+  type: elastic
+icons:


System integration icon might fit better for this case. @mlunadia WDYT? Do we have anything with System Integration + OTel

Or maybe the elastic icon+OTel?

Thanks both for suggestion, just for context, I have used OpenTelemetry and ECS icon here. ECS icon was taken from standard Elastic Icon assets (see https://brand.elastic.co/302f66895/p/031452-icons/b/90604e and check ECS)

kaiyan-sheng

Could you add sample documents for each data stream to test the ingest pipelines? It should be under for example aws_cloudtrail/_dev/test/pipeline/test-aws-cloudtrail.json.

Also just curious why aws_elblogs has suffix logs but the other two dont? 😬

packages/aws/changelog.yml

packages/otel_ecs_converter/_dev/build/docs/README.md

packages/otel_ecs_converter/changelog.yml

Kavindu-Dodan · 2025-10-18T00:04:36Z

Could you add sample documents for each data stream to test the ingest pipelines? It should be under for example aws_cloudtrail/_dev/test/pipeline/test-aws-cloudtrail.json.

Also just curious why aws_elblogs has suffix logs but the other two dont? 😬

Thanks, good suggestion on sample documents. I will add them with the next PR.

Regarding naming, thi s matches existing AWS integration assets. ELB has both logs and metrics support. But for VPC and CloudTrail, there's only logs.

packages/aws/data_stream/vpcflow/elasticsearch/ingest_pipeline/default.yml

...ages/otel_ecs_converter/data_stream/aws_cloudtrail/elasticsearch/ingest_pipeline/default.yml

efd6 · 2025-10-19T21:00:36Z

...ages/otel_ecs_converter/data_stream/aws_cloudtrail/elasticsearch/ingest_pipeline/default.yml

+on_failure:
+- set:
+    field: error.message
+    value: '{{ _ingest.on_failure_message }}'


Suggested change

on_failure:

- set:

field: error.message

value: '{{ _ingest.on_failure_message }}'

- set:

field: event.kind

value: pipeline_error

tag: set_pipeline_error_into_event_kind

if: ctx.error?.message != null

- append:

field: tags

value: preserve_original_event

allow_duplicates: false

if: ctx.error?.message != null

on_failure:

- append:

field: error.message

value: >-

Processor '{{{ _ingest.on_failure_processor_type }}}'

{{{#_ingest.on_failure_processor_tag}}}with tag '{{{ _ingest.on_failure_processor_tag }}}'

{{{/_ingest.on_failure_processor_tag}}}failed with message '{{{ _ingest.on_failure_message }}}'

- set:

field: event.kind

tag: set_pipeline_error_to_event_kind

value: pipeline_error

- append:

field: tags

value: preserve_original_event

allow_duplicates: false

This is a nice suggestion. But I adapted it with some changes as we do not have an original event with OTel. When record reach the pipeline, we already have processed attributes and no original event.

See 1ce7062 and let me know we can agree on this :)

@efd6 let me know your opinion on this :)

packages/otel_ecs_converter/data_stream/aws_cloudtrail/manifest.yml

packages/otel_ecs_converter/data_stream/aws_vpcflow/elasticsearch/ingest_pipeline/default.yml

elasticmachine · 2025-10-20T12:01:09Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

Signed-off-by: Kavindu Dodanduwa <[email protected]> # Conflicts: # packages/aws/changelog.yml # packages/aws/manifest.yml

Signed-off-by: Kavindu Dodanduwa <[email protected]>

Co-authored-by: kaiyan-sheng <[email protected]>

Signed-off-by: Kavindu Dodanduwa <[email protected]>

elasticmachine · 2025-10-21T21:26:02Z

💚 Build Succeeded

Buildkite Build
Commit: 70ba55f

History

💔 Build #32944 failed 70ba55f
💔 Build #32915 failed 1ce7062
💔 Build #32905 failed ee9fb90
💔 Build #32830 failed 64bf7ae
💚 Build #32824 succeeded 4bd7b85

Kavindu-Dodan · 2025-10-22T14:13:26Z

@kaiyan-sheng I looked into adding integration tests, but that requires me to define fields.yml matching all derived fields of the target target data sets. Given this is a converter from OTel to ECS, I do not we should define those target fields inside this component. So I will at least skip the tests in the initial tech preview version release.

andrewkroh added New Integration Issue or pull request for creating a new integration package. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:aws AWS labels Oct 6, 2025

Kavindu-Dodan force-pushed the feat/otel-convertor-pipelines branch 2 times, most recently from c94876a to 2abf6cf Compare October 8, 2025 14:35

Kavindu-Dodan force-pushed the feat/otel-convertor-pipelines branch 3 times, most recently from a72106b to 6ede16d Compare October 14, 2025 17:38

Kavindu-Dodan added the enhancement New feature or request label Oct 14, 2025

Kavindu-Dodan force-pushed the feat/otel-convertor-pipelines branch from 6ede16d to a67d742 Compare October 14, 2025 20:22

Kavindu-Dodan changed the title ~~[WIP] feat: OTel to ECS convertor pipelines~~ [WIP] feat: [AWS] OTel to ECS convertor pipelines Oct 14, 2025

Kavindu-Dodan changed the title ~~[WIP] feat: [AWS] OTel to ECS convertor pipelines~~ [WIP] feat: [AWS] OTel to ECS converter pipelines Oct 14, 2025

Kavindu-Dodan force-pushed the feat/otel-convertor-pipelines branch 2 times, most recently from e912453 to 2c5186a Compare October 16, 2025 18:07

Kavindu-Dodan changed the title ~~[WIP] feat: [AWS] OTel to ECS converter pipelines~~ feat: [AWS] OTel to ECS converter pipelines Oct 16, 2025

Kavindu-Dodan force-pushed the feat/otel-convertor-pipelines branch from 5ad03c2 to 1046b11 Compare October 16, 2025 21:39

Kavindu-Dodan marked this pull request as ready for review October 17, 2025 20:53

Kavindu-Dodan requested review from a team as code owners October 17, 2025 20:53

Kavindu-Dodan force-pushed the feat/otel-convertor-pipelines branch from 1046b11 to 4bd7b85 Compare October 17, 2025 20:55

vigneshshanmugam reviewed Oct 17, 2025

View reviewed changes

kaiyan-sheng reviewed Oct 17, 2025

View reviewed changes

efd6 reviewed Oct 19, 2025

View reviewed changes

andrewkroh added Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Oct 20, 2025

Kavindu-Dodan force-pushed the feat/otel-convertor-pipelines branch from bd5e27b to 1ce7062 Compare October 20, 2025 18:10

Kavindu-Dodan requested a review from efd6 October 21, 2025 14:10

Kavindu-Dodan and others added 14 commits October 21, 2025 07:10

chore: otel to ecf convertor pipeline

0640e2b

Signed-off-by: Kavindu Dodanduwa <[email protected]> # Conflicts: # packages/aws/changelog.yml # packages/aws/manifest.yml

feat: ELB routing and pipeline

f98731a

Signed-off-by: Kavindu Dodanduwa <[email protected]>

chore: add code owners

6670410

Signed-off-by: Kavindu Dodanduwa <[email protected]>

feat: add CloudTrail logs

699995a

Signed-off-by: Kavindu Dodanduwa <[email protected]>

chore: rename and enhance

81d926c

Signed-off-by: Kavindu Dodanduwa <[email protected]>

chore: add todo with reference to OTel issue

e50fd04

Signed-off-by: Kavindu Dodanduwa <[email protected]>

set aws prefix for cloudtrail folder

1a5136e

Signed-off-by: Kavindu Dodanduwa <[email protected]>

use pre_extracted field to check pre-filled status

8076fd2

Signed-off-by: Kavindu Dodanduwa <[email protected]>

Update packages/aws/changelog.yml

2c57954

Co-authored-by: kaiyan-sheng <[email protected]>

Update packages/otel_ecs_converter/_dev/build/docs/README.md

2a8767e

Co-authored-by: kaiyan-sheng <[email protected]>

Update packages/otel_ecs_converter/_dev/build/docs/README.md

be2f6c4

Co-authored-by: kaiyan-sheng <[email protected]>

Update packages/otel_ecs_converter/changelog.yml

a64be58

Co-authored-by: kaiyan-sheng <[email protected]>

document generate

66b5746

Signed-off-by: Kavindu Dodanduwa <[email protected]>

review changes

70ba55f

Signed-off-by: Kavindu Dodanduwa <[email protected]>

Kavindu-Dodan force-pushed the feat/otel-convertor-pipelines branch from 1ce7062 to 70ba55f Compare October 21, 2025 14:10

Kavindu-Dodan requested review from kaiyan-sheng and vigneshshanmugam October 22, 2025 13:42

-on_failure:
-- set:
-    field: error.message
-    value: '{{ _ingest.on_failure_message }}'
+  - set:
+      field: event.kind
+      value: pipeline_error
+      tag: set_pipeline_error_into_event_kind
+      if: ctx.error?.message != null
+  - append:
+      field: tags
+      value: preserve_original_event
+      allow_duplicates: false
+      if: ctx.error?.message != null
+on_failure:
+  - append:
+      field: error.message
+      value: >-
+        Processor '{{{ _ingest.on_failure_processor_type }}}'
+        {{{#_ingest.on_failure_processor_tag}}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
+        {{{/_ingest.on_failure_processor_tag}}}failed with message '{{{ _ingest.on_failure_message }}}'
+  - set:
+      field: event.kind
+      tag: set_pipeline_error_to_event_kind
+      value: pipeline_error
+  - append:
+      field: tags
+      value: preserve_original_event
+      allow_duplicates: false

feat: [AWS] OTel to ECS converter pipelines #15570

Are you sure you want to change the base?

feat: [AWS] OTel to ECS converter pipelines #15570

Conversation

Kavindu-Dodan commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Footnotes

Uh oh!

elastic-vault-github-plugin-prod bot commented Oct 9, 2025

🚀 Benchmarks report

Uh oh!

vigneshshanmugam Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

kaiyan-sheng Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

Kavindu-Dodan Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaiyan-sheng left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Kavindu-Dodan commented Oct 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Kavindu-Dodan Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kavindu-Dodan Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Oct 20, 2025

Uh oh!

elasticmachine commented Oct 21, 2025

💚 Build Succeeded

History

Uh oh!

Kavindu-Dodan commented Oct 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Kavindu-Dodan commented Oct 6, 2025 •

edited

Loading

Kavindu-Dodan Oct 17, 2025 •

edited

Loading

Kavindu-Dodan Oct 20, 2025 •

edited

Loading